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One  of  the  "4^  problems  Identified  by  the  Director  of  Defense 
Research  and  Engineering,  in  his  high-priority  effort  to 
strengthen  the  Department  of  Defense  in-house  laboratories, 
concerned . the  evaluation  of  research  and  development  organi¬ 
zations.^'^  This  report  describes  a  peer  rating  experiment 
which  was  an  attempt  to  gain  greater  insight  into  the  rela¬ 
tionship  of  management  and  organizational  performance  with 
organizational  characteristics. 


^Problms  of  the  In-Houee  Laboratories  and  Poeeible 
Solutions  (Washington,  D.C. :  Office  of  the  Director  of 
Defense  Research  and  Engineering,  Management  Analysis  Memo- 
randtsD  (MAM)  66-3,  25  October  1966). 
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INTRODUCTION 


R&D  managers  have  been  trying  for  many  years  to  devise  better  yard¬ 
sticks  to  measure  the  effectiveness  and  utility  of  laboratories.  Most 
managers  agree  that  such  techniques  are  needed,  but  few  can  agree  on  how 
much  appraisals  can  or  should  be  made.  Both  "hard"  and  "sdft"  scientists 
have  made  many  attempts  to  "get  a  handle"  on  this  question,  from  physi¬ 
cist  and  Nobel  laureate  William  Shockley,  in  his  paper,  "On  Statistics 
of  Individual  Variations  of  Productivity  in  Research  Laboratories,"  to 
social  scientist  Professor  Donald  C.  Pels,  in  his  study,  "Motivations 
and  Working  Relations  of  Scientists  and  Engineers." 

During  the  period  1966-1968,  interest  in  Improving  the  effective¬ 
ness  of  Federal  laboratories  heightened.  It  was  during  that  time  that 
the  Congress  became  Interested  in  the  mattex  of  Federal  laboratory  per¬ 
formers.  Both  the  Subcommittee  on  Research  and  Technical  Programs 
(Reuss)  of  the  House  Committee  on  Government  Operations  and  the  Subcom¬ 
mittee  on  Science,  Research  and  Development  (Daddarlo)  of  the  House 
Committee  on  Science  and  Astronautics  recommended  that  greater  attention 
be  given  to  this  Important  area.  Six  days  of  hearings  were  held  by  the 
Daddarlo  Subcommittee  early  in  1968  on  the  utilization  of  Federal  labo¬ 
ratories  . 


OADDARIO  SUBCOMMITTEE  HEARINGS 

A  good  deal  of  attention  was  paid  to  the  appraisal  of  Government 
laboratories  during  these  hearings.  Dr.  Donald  Homlg,  then  scientific 
adviser  to  the  President,  described  the  various  techniques  used  within 
the  Government  to  appraise  laboratories.  He  pointed  out  that,  to  a  con¬ 
siderable  degree,  laboratory  evaluations  are  necessarily  qualitative 
rather  than  quantitative  and  involve  Judgments  based  on  such  factors  as 
experience  and  comparison  with  good  practice  elsewhere. 

Dr.  Allan  Astln,  Director  of  the  National  Bureau  of  Standards,  felt 
that  It  was  essential  to  have  some  mechanism  for  rating  laboratories: 

I  think  one  of  my  responsibilities  is  to  have  techniques  for 
rating  the  different  laboratories  within  the  organization, 
and  where  the  leadership  or  management  is  Ineffective,  doing 
something  about  strengthening  it  as  well  as  encouraging  those 
that  are  strong. 

Or.  William  McLean^  Technical  Director  of  the  Navy  Undersea  Warfare 
Center,  believes  that  a  nonsubj active  measure  of  laboratory  effective¬ 
ness  is  competition  between  laboratories  and  that  Judgment  should  be 
made  on  the  record  of  their  accomplishments.  This  can  only  be  done, 
however,  by  an  appraiser  who  has  been  successful  In  the  field  being 
evaluated. 
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Or.  Harold  Finger,  theu  Associate  Administrator,  Office  of  Organi¬ 
zation  and  Management,  National  Aeronautics  and  Space  Administration 
(NASA) ,  described  the  problem  of  evaluating  the  capability  and  perform¬ 
ance  of  research  organizations  as  a  difficult  one: 

Perhaps  It  Is  one  which  can  be  answered  only  subjectively, 
since  It  Is  difficult  to  quantify  or  score  so  complex  and 
sophisticated  an  organization  as  a  laboratory. 

He  believes  that  NASA  experience  confirms  a  widely  held  view  among  re¬ 
search  administrators  that  the  basic  determinant  of  strength  or  weakness 
Is  the  fundamental  Issue  of  the  value  and  Importance  of  the  organization's 
purposes,  the  validity  of  Its  objectives  and  Its  capability  to  satisfy 
those  purposes  and  objectives. 

It  would  seem  hard  for  a  research  organization  to  succeed  If 
Its  basic  purposes  and  motivations  are  unimportant.  On  the 
other  end,  the  research  organization  which  has  an  Important 
and  significant  role  has  a  good  start  In  the  critical  process 
of  building  and  holding  a  research  competence. 

After  deliberating  over  these  and  many  other  thoughts  on  laboratory 
evaluation,  the  Daddarlo  Subcommittee  concluded: 

The  Subcommittee  believes  that  appraisal  of  Government  Labora¬ 
tories,  both  for  their  scientific  and  their  technical  perform¬ 
ance,  is  an  essential  element  of  Federal  Laboratory  administra¬ 
tion.  ...  It  Is  also  apparent  to  the  Subcommittee  that 
Federal  principles  and  procedures  for  laboratory  appraisal  are 
not  yet  commonly  agreed  upon  nor  is  there  complete  agreement 
that  appraisal  of  scientific  productivity  and  performance  Is 
feasible.  Nonetheless  agency  heads  responsible  for  major  In¬ 
vestments  In  Government  Laboratories  cannot  wait  until  the 
Ideal  or  final  method  Is  found.  They  must  go  ahead  with  what 
is  now  available  and  Improve  it  through  use. 


DOD  INTEREST  IN  APPRAISAL 

The  Department  of  Defense  (DoD)  employs  about  60  percent  of  the 
civil  service  engineers  and  35  percent  of  the  civil  service  scientists 
In  the  Federal  Government,  most  of  whom  work  in  laboratories  and  test 
and  evaluation  activities.  In  FY  1968,  RDT&E  (research,  development, 
test  and  evaluation)  obligations  for  these  laboratories  were  $1.8  billion; 
about  half  of  this  was  utilized  to  support  In-house  work,  the  remainder 
being  contracted  out.  These  laboratories  employ  about  72,000  people,  of 
whom  about  28,000  are  scientists  and  engineers.  Based  upon  acquisition 
costs,  the  investment  In  physical  plant  and  equipment  Is  about  $2.2  bil¬ 
lion. 
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This  Is  a  major  Investment,  fnd  we  are  anxlotis  to  see  that  It  Is 
managed  judiciously  and  effectively  and  that  It  Is  used  to  solve  the 
pressing  problems  of  the  three  Military  Departments  and  the  six  Defense 
Agencies.  This  requires  appraisals  and  tough-minded  decisions  concerning 
laboratories  with  poor  or  marginal  capabilities  and  management  encourage¬ 
ment  for  laboratories  of  high  quality. 

Within  the  DoD  many  different  types  of  appraisals  are  regularly 
made — supervisory  evaluations,  program  evaluations,  special  appraisals, 
committee  visits,  and  the  natural  competition  of  laboratories  for  Impor¬ 
tant  programs .  Most  of  these  techniques  are  subjective  In  nature  and 
lack  a  quantitative  basis,  particularly  for  comparisons  among  laborato¬ 
ries  with  widely  differing  missions  and  technical  orientation. 

To  rectify  some  of  the  deficiencies  in  the  current  appraisal  systems, 
a  comprehensive  data  base  has  been  developed  to  provide  comparative  sta¬ 
tistical  and  trend  data  on  the  characteristics  and  performance  of  labora¬ 
tories.  Yet  this  In  Itself  was  not  considered  entirely  adequate.  It  was 
felt  that  the  utility  and  significance  of  these  data  might  be  Improved  If 
we  could  relate  them  somehow  to  the  comparative  technical  competence  or 
quality  of  laboratories.  In  seeking  a  relatively  simple  method  of  rank¬ 
ing  laboratories  by  quality,  we  elected  to  use  an  Apstein-modified  Pelz 
technique. ^ 

Apsteln-Modlfled  Pelz  Technique 

From  1960  to  1963,  one  of  the  top  administrators  of  the  Army's  Harry 
Diamond  Laboratories,  Or.  Maurice  Apstein,  an  engineer  by  profession, 
made  a  study  of  "The  Role  of  the  Military  Laboratory  in  Defense"  as  his 
doctoral  thesis  at  the  American  University.  He  attempted  to  assess  the 
relative  competence  of  a  selected  group  of  Government  laboratories  so 
that  he  could  examine  the  relationship  of  quality  with  other  lab  charac¬ 
teristics.  He  selected  and  modified  a  procedure  used  by  Pelz  In  his 
study  of  professional  relationships  within  laboratories.^ 

The  technique  consists  of  a  series  of  carefully  controlled  peer 
ratings  In  which  a  rank  order  Is  arrived  at  by  means  of  a  sequence  of 
paired  comparisons.  The  peers,  or  judges,  consist  of  professional  tech¬ 
nical  people  with  a  substantial  degree  of  Industrial,  university  or 
Federal  laboratory  experience,  mostly  in  the  management  of  R&D  programs 
and  organizations.  Emphasis  Is  placed  upon  the  technical  rather  than 
the  administrative  background  of  the  individual  judge  so  that.  In  his 

^Maurice  Apstein,  "Effectiveness  of  Military  Laboratories  as  a 
Function  of  Contract  Activity,"  IEEE  Traneaoticms  on  Engineering  Manage- 
ment,  EM-12,  No.  2,  June  1965,  pp.  44-50. 

^D.  C.  Pelz,  Motivations  and  Working  Relations  of  Scientists  and 
Engineers  (Ann  Arbor,  Michigan:  Institute  of  Social  Research,  University 
of  Michigan,  Preliminary  Report  4,  June  1960). 
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judgment  of  a  given  organization)  consideration  would  be  given  more  to 
technical  competence  than  to  administrative  efficiency. 

Methodology 

The  name  and  location  of  each  DoD  labcvatory  were  printed  on  Indi¬ 
vidual  cards.  Each  judge  was  given  a  set  of  cards  with  the  following 
Instructions: 

1)  Here  Is  a  list  of  the  major  R  and  D  Installations  In  the 
Department  of  Defense.  Please  separate  them  Into  two  piles; 
those  you  know  and  those  you  do  not.  For  purposes  of  this 
exercise,  to  "know"  a  laboratory  le  defined  as  being  suffici¬ 
ently  acquainted  wl th  Its  work  to  have  formed  an  opinion 
regarding  Its  technical  competence  to  perform  Its  assigned 
mission.  This  opinion  need  not  have  been  obtained  first  hand; 

It  may  have  been  formed  through  reading  government  reports, 
technical  articles  In  the  open  literature,  and  via  Inputs 
from  other  scientific  professionals  whose  judgment  you  respect. 

If  there  Is  any  question  In  your  mind  regarding  the  validity  of 
your  Information,  place  the  card  In  the  "unknown"  pile. 

2)  Discard  the  "unknown"  pile  and  separate  the  known  pile  Into 
three  groups,  GOOD,  MEDIUM,  and  POOR. 

3)  Now  take  the  GOOD  pile  and  lay  the  cards  In  front  of  you  so 
that  they  are  all  In  view.  Place  them  In  ruiklng  order  by  se¬ 
lecting  first  the  BEST  of  the  group,  then  the  next  best,  and  so 
on  until  you  have  ranked  the  entire  group.  Place  this  pile 
aside. 

4)  Repeat  with  the  MEDIUM  pile. 

5)  Repeat  with  the  BELOW  AVERAGE  and  POOR  groups. 

6)  Combine  all  the  plies  In  ranking  order. 

It  has  been  claimed  that  the  survey  does  not  really  measure  techni¬ 
cal  competence  but  Instead  measures  technical  reputation.  Apsteln's 
assustptlon  was  that  technical  reputation  In  the  scientific  community  la 
based  upon  quality  of  scientific  work.  The  two  terms  were  therefore 
considered  synonymous. 

The  conclusions  of  the  Apsteln  study  were  as  follows: 

1)  A  system  of  peer  ratings  and  paired  comparisons  appears  to 
permit  the  ranking  of  selected  R  and  D  laboratories  In  order 
of  technical  competence. 

2)  There  Is  a  discernible  relationship  between  technical  compe¬ 
tence  and  the  size  of  the  contract  effort  administered  by  the 
laboratory. 
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3)  A  contract  program  greater  than  60  percent  of  the  total 
laboratory  budget  appears  to  represent  an  administrative 
burden,  which  impairs  the  cranpetence  of  the  technical  staff. 

4)  A  laboratory  which  does  an  insignificant  amount  of  contract 
supervision  should  take  special  care  to  Insure  Interaction 
with  the  rest  of  the  scientific  conmunlty. 

Current  Study 

The  current  study  by  Mr.  Evan  D.  Anderson,  Office  for  Laboratory 
Management,  ODDR&E,  expands  upon  the  original  Apsteln  method  in  several 
ways.  A  tenfold  Increase  in  judges  Is  being  adopted,  and  participants 
represent  a  much  broader  spectrum  of  technical  people.  The  judges  se¬ 
lected  can  be  categorized  in  five  groups: 

(1)  DoD  laboratory  directors 

(2)  R&D  managers  and  technical  specialists  within  the 

Federal  Government 

(3)  Industrial  technical  specialists,  consultants  and 

professionals  from  nonprofit  organizations 

(4)  Scientists  and  engineers  in  academic  institutions 

(5)  Technical  specialists  in  DoD  program  management  and 

system  project  offices 

This  enables  us  to  examine  the  ratings  of  laboratories  from  many 
different  perspectives  and  points  of  view.  Of  particular  Interest  are 
the  attitudes  of  "customers"  of  the  laboratories.  These  are  the  techni¬ 
cal  personnel  in  the  program  management  and  system  project  offices  who 
utilize  the  services  of  many  Defense  laboratories  and  are  able  to  judge 
them  on  the  basis  of  fairly  specific  performance  factors. 

We  have  obtained  ratings  from  about  250  judges  thus  far,  and  hope 
ultimately  to  have  about  400  people  ranking  the  laboratories.  Selected 
examples  of  the  preliminary  data  are  described  in  Table  1  and  Figure  1. 

The  rankings  have  been  normalized  by  arrangement  on  a  percentage 
basis,  using  the  decile  as  the  class  Interval.  Examples- of  high-, 
medium-  and  low-ranking  laboratories  are  shown.  Six  laboratories  with 
staffs  (N)  of  at  least  75  were  selected  for  comparison.  The  distribu¬ 
tion  shows  consistent  patterns  as  one  proceeds  from  the  highest  rated 
(laboratory  rank  No.  1)  to  the  lowest  in  this  subset  (laboratory  rank 
No.  56).  They  are  similar  to  the  cl5stributionp  plotted  by  Apsteln,  al¬ 
though  he  utilized  a  much  smaller  sample  of  judges.  The  statistical 
significance  of  the  difference  in  ratings  will  be  determined  when  the 
total  rating  sample  has  been  analyzed.  It  is  not  our  Intent  to  develop 
a  precise  rank  ordering,  but  to  use  these  rankings  as  a  means  of  ex¬ 
ploring  the  relationships  between  technical  reputation  and  the  many 
measurable  characteristics  of  laboratories. 
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The  Future 


Social  scientists  might  argue  that  this  experiment  Is  much  too 
simple  a  design  to  yield  accurate  knowledge  and  Insight  into  management 
and  organizational  performance.  Tet  we  believe  It  is  at  present  a  use¬ 
ful  adjunct  to  other  appraisal  techniques  we  currently  employ.  Hopeful¬ 
ly,  the  kind  of  organizational  research  recently  described  by  Likert  and 
Bowers'*  will  provide  more  meaningful  answers  to  the  complex  relation¬ 
ships  among  organizational  variables. 

The  first  phase  of  this  study  will  consist  of  analyses  based  on 
extensive  data  banks  already  developed  on  laboratory  operations.  From 
that  we  hope  to  understand  better  why  certain  laboratories  are  rated  high 
while  others  are  given  low  ratings.  Is  It  geographical  location  or  loca¬ 
tion  near  or  distant  from  a  "rich"  academic  community?  Do  their  posi¬ 
tions  In  the  organizational  hierarchy  matter?  Is  organizational  size  or 
level  of  funding  a  factor?  Is  their  orientation  toward  research  or  hard¬ 
ware  development  relevant?  Uhat  about  their  relative  levels  of  In-house 
work  versus  work  contracted  out?  Are  such  output  Indicators  as  patents, 
papers,  reports,  etc.,  meaningful?  Do  staff  age,  advanced  degrees, 
discipline  mixes,  educational  opportunities,  salary  levels,  turnover 
rates,  etc.,  show  Important  relationships  with  quality  ranking? 

We  plan  to  utilize  multiple  variate  analysis  to  answer  these'and 
similar  questions  to  gain  deeper  insight  Into  the  causes  of  variation 
between  different  samples  of  scientists  and  engineers.  Hopefully,  this 
will  provide  a  better  understanding  of  the  similarities  and  differences 
of  those  laboratories  rated  high  in  technical  reputation  and  those  that 
receive  lower  ratings. 

The  second  phase  of  the  study  Is  expected  to  encompass  a  much  more 
meaningful  attempt  to  redesign  or  supplement  present  criteria  measures 
to  increase  understanding,  reliability  and  validity.  This  would  extend 
the  range  of  variable.^  studied  to  Include  consideration  of  reporting  and 
control  procedures,  management  policies,  organization  structure,  profes¬ 
sional  attitudes,  leadership  patterns,  etc.  An  analysis  of  these  addi¬ 
tional  variables  would  provide  more  thorough  understanding  of  the  man¬ 
agement  actions  and  policies  that  Influence  laboratory  productivity. 

The  ultimate  goal  Is  to  give  the  managers  of  DoD  laboratories  a 
greater  Insight  Into  research  management  and  organization.  It  will  help 
them  In  effectively  using  data  on  laboratory  properties,  performance  and 
their  relationships,  for  purposes  of  self-evaluation  and  self-improvement. 
Finally,  we  believe  It  can  assist  top  management  in  formulating  relevant 
policies  and  practices  that  will  create  the  required  organizational  en¬ 
vironment  for  the  DoD  laboratories. 

'*R.  Likert  and  D.  G.  Bowers,  "Organizational  Iheory  and  Human 
Resources  Accounting,"  Annual  Meeting  of  the  American  Psychological 
Association,  San  Francisco,  California,  September  1968. 
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